Poll for myCnf in the MySQL connection-timeout CLI tests - #1192

Failures fixed (investigator)

• source: Percona-Lab/pmm-submodules#4513 — run 32030290448, check CLI / Integration tests / CLI / Integration / Percona Server 8.4 (job 95388558456)
• tests:
  • cli/tests/perconaMySqlServer.spec.ts:180 / @percona-server — PMM-T2221 "User can use connection timeout while using pmm-admin add"

What failed

Percona Server 8.4 was one of two red checks in that run. Its Run CLI tests step ends in
|| true, so the job went red one step later on launchable gate (Actionable Failures | 1):

Error: Stdout does not contain connect_timeout=5, !

expect(received).toContain(expected) // indexOf
Expected substring: "connect_timeout=5"
Received string:    ""
  at cli/tests/perconaMySqlServer.spec.ts:190:5

The received string is empty — the cat …/myCnf produced no output at all.

The same 23 tests, same FB image, passed on Percona Server 5.7 and Percona Server 8.0 in
that same run. The telling detail is the timing: PMM-T2221 took 2.4s on the leg that failed
and 3.6s on both legs that passed.

Root cause — the test races an asynchronous write

pmm-admin add returns as soon as the server has recorded the agent. pmm-agent writes the
mysqld_exporter myCnf only once it receives the resulting state, some time later. The test
covered that gap with a fixed sleep 2 and then read the file exactly once:

await output.exitCodeEquals(0);
await cli.exec('sleep 2');
… // cat pmm-agent.yaml, pmm-admin list, pmm-admin list
const myCnf = await cli.exec(`docker exec … cat ${tempDir}/…/${agentId}/myCnf`);
await myCnf.outContains('connect_timeout=5');

So the real budget is 2s plus however long those three intervening docker exec calls take —
which is why the leg where they were faster (2.4s total) is the one that failed, while the two
slower legs (3.6s) passed. A test that gets less reliable as the machine gets faster is a race,
not a product problem.

Measured directly on the FB build (perconalab/pmm-server-fb:PR-4513-8106301, digest
sha256:8b323a74… — the same digest Launchable recorded for the failing session), on an idle
6-core box, polling every 100 ms:

run 1: add_cmd=0.15s  myCnf_after_add=1.36s
run 2: add_cmd=0.13s  myCnf_after_add=1.17s
run 3: add_cmd=0.14s  myCnf_after_add=1.15s
run 4: add_cmd=0.14s  myCnf_after_add=1.20s
run 5: add_cmd=0.16s  myCnf_after_add=1.19s

The file does not exist when pmm-admin add returns; it appears ~1.2s later and does contain
connect_timeout=5. PMM is doing the right thing — the 2s margin is simply not a guarantee.
PMM-T2222 and PMM-T2223 read the same file the same way, behind sleep 5.

The fix

Replace the fixed sleeps + one-shot read with a polling assertion (expect(…).toPass, the
pattern already used in mongoDb-psmdb.spec.ts) that re-resolves the agent id and re-reads
myCnf every second for up to 60s. The service id / agent id lookups moved inside the poll too,
so a not-yet-listed agent retries instead of silently producing a bad path.

Nothing is loosened: the assertion is still the exact same substring (connect_timeout=5 / =4
/ =2). A build that stops writing the connection timeout still fails this test — it just now
fails after 60s of trying rather than at an arbitrary 2s mark.

Verification

Reproduced and fixed on a throwaway Linode VM following runner-integration-cli-tests.yml, with
the failing run's own PMM_SERVER_IMAGE=perconalab/pmm-server-fb:PR-4513-8106301,
CLIENT_VERSION=…pmm-client-PR-4513-8106301.tar.gz and --database ps=8.4.

• Baseline — the suite is green here as-is on a fast idle box (23 passed), matching the two
  PS legs that passed in CI. Reproducing the red needs the race widened.

• A/B under the same contention (docker update --cpus 0.3 pmm-server, everything else
  identical, 3 runs each):

  spec	result
  main (a2bf067)	2 failed / 1 passed — failures identical to CI: Expected substring: "connect_timeout=5" / Received string: ""
  this branch	3 passed (12.0s, 12.8s, 6.3s)

• Full @percona-server suite on this branch against the FB image, server unthrottled — the
  same 23 tests CI ran: 23 passed (25.0s). PMM-T2222/T2223 also got faster (3.1s / 2.2s vs
  ~5.9s), since they no longer sit through a fixed 5s sleep.

• Not blocked on anything upstream. The change is test-side only and passes against the FB
  image and against main's behaviour alike — normal PR, not a draft.

• npm run lint (eslint + tsc --noEmit) in cli/: Lint OK, 0 errors.

The 60s poll budget is generous against a measured ~1.2s, but it is still a bound: at an
absurd --cpus 0.1 throttle (where a single pmm-admin list takes seconds) I saw one run
exceed it. That is far past any CI runner and I did not tune for it.

Not covered by this PR

The other red check in that run, E2E / Docker configuration tests / e2e tests: @docker-configuration (PMM-T2237, all three ClickHouse configs), is already tracked by the open
PR #1180 — untouched here.

The same fixed-sleep-then-read-once pattern exists in the PostgreSQL, MongoDB, Valkey and
ProxySQL variants of PMM-T2221/T2222 (they grep pmm-agent.log rather than myCnf). They did
not fail in this run and are left alone; worth the same treatment if they start flaking.

Generated by Claude Code